Skip to content

Add test reporting for Karate v2#11923

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 3 commits into
masterfrom
daniel.mohedano/karate-v2-instrumentation
Jul 14, 2026
Merged

Add test reporting for Karate v2#11923
gh-worker-dd-mergequeue-cf854d[bot] merged 3 commits into
masterfrom
daniel.mohedano/karate-v2-instrumentation

Conversation

@daniel-mohedano

@daniel-mohedano daniel-mohedano commented Jul 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What Does This Do

  • Adds a new karate-2.0 instrumentation module to implement test visibility for Karate 2.0+.
    • The implementation follows the same approach used for the original instrumentation in the karate-1.0 module.
    • Also implements skip-based features (ITR and disabled tests), as part of the tracing listener.

Motivation

Karate v2 is a complete ground-up rewrite of the framework. Because of this, the original karate-1.0 module cannot instrument it.

Additional Notes

A following PR will contain the changes regarding execution instrumentation for Karate v2.

Contributor Checklist

Jira ticket: SDTEST-3816

@daniel-mohedano daniel-mohedano added type: feature Enhancements and improvements comp: ci visibility Continuous Integration Visibility labels Jul 13, 2026
@daniel-mohedano daniel-mohedano force-pushed the daniel.mohedano/karate-v2-instrumentation branch from 4d2d28e to 1d0e84d Compare July 13, 2026 08:06
@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 13, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Environment - sbt-scalatest

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 57.49 55.43 $\color{red}{\blacktriangle}$ +2.06 54.33 $\color{red}{\blacktriangle}$ +3.16 76/242
agentEvpProxy 55.39 n/a n/a n/a n/a -

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 13, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Environment - reactive-streams-jvm

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 21.62 22.09 $\color{green}{\blacktriangledown}$ -0.47 21.65 $\color{green}{\blacktriangledown}$ -0.03 39/126
agentless 18.55 18.82 $\color{green}{\blacktriangledown}$ -0.27 18.82 $\color{green}{\blacktriangledown}$ -0.27 38/123
agentlessCodeCoverage 19.83 20.39 $\color{green}{\blacktriangledown}$ -0.56 19.99 $\color{green}{\blacktriangledown}$ -0.16 38/122
agentlessLineCoverage 30.28 29.82 $\color{red}{\blacktriangle}$ +0.46 29.82 $\color{red}{\blacktriangle}$ +0.46 37/121

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 13, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Environment - nebula-release-plugin

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 37.15 37.15 $\color{gray}{\bullet}$ 0.00 36.42 $\color{red}{\blacktriangle}$ +0.73 38/119
agentless 38.43 36.42 $\color{red}{\blacktriangle}$ +2.01 36.42 $\color{red}{\blacktriangle}$ +2.01 38/119
agentlessCodeCoverage 45.10 45.38 $\color{green}{\blacktriangledown}$ -0.28 44.48 $\color{red}{\blacktriangle}$ +0.62 38/119
agentlessLineCoverage 75.01 74.82 $\color{red}{\blacktriangle}$ +0.19 74.82 $\color{red}{\blacktriangle}$ +0.19 37/118

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 13, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Environment - pass4s

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 9.64 9.35 $\color{red}{\blacktriangle}$ +0.29 10.33 $\color{green}{\blacktriangledown}$ -0.69 38/118
agentless 10.33 10.33 $\color{gray}{\bullet}$ 0.00 10.13 $\color{red}{\blacktriangle}$ +0.20 38/118
agentlessCodeCoverage 16.75 17.03 $\color{green}{\blacktriangledown}$ -0.28 17.38 $\color{green}{\blacktriangledown}$ -0.63 37/116

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 13, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Environment - heliboard

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 10.88 9.54 $\color{red}{\blacktriangle}$ +1.34 9.54 $\color{red}{\blacktriangle}$ +1.34 36/40

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 13, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Environment - sonar-kotlin

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 13.24 12.87 $\color{red}{\blacktriangle}$ +0.37 13.13 $\color{red}{\blacktriangle}$ +0.11 34/117
agentless 12.65 11.88 $\color{red}{\blacktriangle}$ +0.77 12.12 $\color{red}{\blacktriangle}$ +0.53 34/116
agentlessCodeCoverage 15.67 15.11 $\color{red}{\blacktriangle}$ +0.56 15.11 $\color{red}{\blacktriangle}$ +0.56 34/116
agentlessLineCoverage 19.37 19.20 $\color{red}{\blacktriangle}$ +0.17 19.20 $\color{red}{\blacktriangle}$ +0.17 33/116

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

@datadog-official

This comment has been minimized.

Sign in to view
@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 13, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Environment - jolokia

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 92.27 95.12 $\color{green}{\blacktriangledown}$ -2.85 93.23 $\color{green}{\blacktriangledown}$ -0.96 41/128
agentless 88.34 89.58 $\color{green}{\blacktriangledown}$ -1.24 89.58 $\color{green}{\blacktriangledown}$ -1.24 39/124
agentlessCodeCoverage 98.30 99.00 $\color{green}{\blacktriangledown}$ -0.70 99.00 $\color{green}{\blacktriangledown}$ -0.70 40/123
agentlessLineCoverage 99.44 99.00 $\color{red}{\blacktriangle}$ +0.44 99.00 $\color{red}{\blacktriangle}$ +0.44 38/121

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 13, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Environment - netflix-zuul

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 86.69 87.80 $\color{green}{\blacktriangledown}$ -1.11 87.80 $\color{green}{\blacktriangledown}$ -1.11 39/126
agentless 80.90 81.05 $\color{green}{\blacktriangledown}$ -0.15 81.05 $\color{green}{\blacktriangledown}$ -0.15 39/125
agentlessCodeCoverage 96.48 97.04 $\color{green}{\blacktriangledown}$ -0.56 95.12 $\color{red}{\blacktriangle}$ +1.36 39/123
agentlessLineCoverage 114.34 111.62 $\color{red}{\blacktriangle}$ +2.72 111.62 $\color{red}{\blacktriangle}$ +2.72 38/122

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 13, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Environment - okhttp

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 20.52 19.20 $\color{red}{\blacktriangle}$ +1.32 19.20 $\color{red}{\blacktriangle}$ +1.32 40/130
agentless 18.01 19.20 $\color{green}{\blacktriangledown}$ -1.19 19.20 $\color{green}{\blacktriangledown}$ -1.19 39/127
agentlessCodeCoverage 22.86 22.99 $\color{green}{\blacktriangledown}$ -0.13 22.09 $\color{red}{\blacktriangle}$ +0.77 38/125
agentlessLineCoverage 44.09 44.48 $\color{green}{\blacktriangledown}$ -0.39 44.48 $\color{green}{\blacktriangledown}$ -0.39 37/131

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 13, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Environment - spring_boot

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 16.43 16.36 $\color{red}{\blacktriangle}$ +0.07 16.04 $\color{red}{\blacktriangle}$ +0.39 38/117
agentless 9.68 9.73 $\color{green}{\blacktriangledown}$ -0.05 9.73 $\color{green}{\blacktriangledown}$ -0.05 38/118
agentlessCodeCoverage 13.65 13.67 $\color{green}{\blacktriangledown}$ -0.02 13.40 $\color{red}{\blacktriangle}$ +0.25 39/117
agentlessLineCoverage 32.89 32.95 $\color{green}{\blacktriangledown}$ -0.06 32.95 $\color{green}{\blacktriangledown}$ -0.06 37/116

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

@dd-octo-sts

dd-octo-sts Bot commented Jul 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.75 s 14.60 s [+0.2%; +1.9%] (maybe worse)
startup:insecure-bank:tracing:Agent 13.55 s 13.63 s [-1.3%; +0.2%] (no difference)
startup:petclinic:appsec:Agent 16.86 s 16.71 s [-0.1%; +1.9%] (no difference)
startup:petclinic:iast:Agent 16.92 s 16.83 s [-0.2%; +1.3%] (no difference)
startup:petclinic:profiling:Agent 16.55 s 16.75 s [-2.4%; +0.0%] (no difference)
startup:petclinic:sca:Agent 16.89 s 16.81 s [-0.7%; +1.8%] (no difference)
startup:petclinic:tracing:Agent 15.63 s 15.96 s [-6.4%; +2.2%] (no difference)

Commit: 49a54e8d · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@daniel-mohedano daniel-mohedano force-pushed the daniel.mohedano/karate-v2-instrumentation branch from 1d0e84d to e7ddda7 Compare July 13, 2026 08:47
@daniel-mohedano daniel-mohedano added the tag: override groovy enforcement Override the "Enforce Groovy Migration" check label Jul 13, 2026
@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 13, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Environment - sonar-java

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent -8.86 12.37 $\color{green}{\blacktriangledown}$ -21.23 13.67 $\color{green}{\blacktriangledown}$ -22.53 36/123
agentless 18.79 13.40 $\color{red}{\blacktriangle}$ +5.39 16.04 $\color{red}{\blacktriangle}$ +2.75 36/122
agentlessCodeCoverage 69.02 77.88 $\color{green}{\blacktriangledown}$ -8.86 86.07 $\color{green}{\blacktriangledown}$ -17.05 36/122
agentlessLineCoverage 139.85 130.99 $\color{red}{\blacktriangle}$ +8.86 136.34 $\color{red}{\blacktriangle}$ +3.51 35/121

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

@daniel-mohedano daniel-mohedano force-pushed the daniel.mohedano/karate-v2-instrumentation branch from 36148cd to 294d720 Compare July 13, 2026 10:22
@daniel-mohedano

daniel-mohedano commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

chatgpt-codex-connector[bot]
chatgpt-codex-connector Bot reviewed Jul 13, 2026
View reviewed changes

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2599ffbfcf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread ...ion/karate/karate-2.0/src/main/java21/datadog/trace/instrumentation/karate2/KarateUtils.java Outdated
Comment thread dd-java-agent/instrumentation/karate/karate-2.0/src/test/groovy/KarateV2Test.groovy Outdated
Comment thread ...ion/karate/karate-2.0/src/main/java21/datadog/trace/instrumentation/karate2/KarateUtils.java Outdated
Comment thread ...ion/karate/karate-2.0/src/main/java21/datadog/trace/instrumentation/karate2/KarateUtils.java Outdated
@daniel-mohedano daniel-mohedano force-pushed the daniel.mohedano/karate-v2-instrumentation branch from 2599ffb to cbdc3d4 Compare July 13, 2026 12:25
@daniel-mohedano daniel-mohedano added the tag: ai generated Largely based on code generated by an AI or LLM label Jul 13, 2026
@daniel-mohedano daniel-mohedano force-pushed the daniel.mohedano/karate-v2-instrumentation branch from cbdc3d4 to e459f4b Compare July 13, 2026 12:32
@daniel-mohedano

daniel-mohedano commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

chatgpt-codex-connector[bot]
chatgpt-codex-connector Bot reviewed Jul 13, 2026
View reviewed changes

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e459f4bc8e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .../karate-2.0/src/main/java21/datadog/trace/instrumentation/karate2/KarateTracingListener.java
Comment thread .../karate-2.0/src/main/java21/datadog/trace/instrumentation/karate2/KarateTracingListener.java
Comment thread ...ion/karate/karate-2.0/src/main/java21/datadog/trace/instrumentation/karate2/KarateUtils.java
Comment thread .../karate-2.0/src/main/java21/datadog/trace/instrumentation/karate2/KarateTracingListener.java
@daniel-mohedano daniel-mohedano mentioned this pull request Jul 13, 2026
Open
@daniel-mohedano daniel-mohedano marked this pull request as ready for review July 13, 2026 14:36
@daniel-mohedano daniel-mohedano requested review from a team as code owners July 13, 2026 14:36
@daniel-mohedano daniel-mohedano requested review from dougqh and vandonr and removed request for a team July 13, 2026 14:36
@daniel-mohedano daniel-mohedano changed the title Implement test visibility for Karate v2 Add test reporting for Karate v2 Jul 13, 2026
datadog-official[bot]
datadog-official Bot reviewed Jul 13, 2026
View reviewed changes

@datadog-official datadog-official Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: PASS

More details

The karate-2.0 instrumentation is behaviourally correct for its stated scope (basic test reporting + ITR skipping). The most non-obvious risk — that SCENARIO_EXIT fires even when the listener vetoes SCENARIO_ENTER — was confirmed safe: Karate v2 populates the result with skipped-step entries, so afterScenario does not spuriously call onTestSkip, and the unconditional onTestFinish call is a no-op because onTestIgnore already removed the test from inProgressTests. Tag names are stored without the @ prefix, confirming the datadog_itr_unskippable guard works correctly.

Was this helpful? React 👍 or 👎

📊 Validated against 4 scenarios · Open Bits AI session

🤖 Datadog Autotest · Commit e459f4b · What is Autotest? · Any feedback? Reach out in #autotest

@daniel-mohedano

daniel-mohedano commented Jul 14, 2026

Copy link
Copy Markdown
Contributor Author

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Jul 14, 2026 •
edited
Loading

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-07-14 15:12:08 UTC ℹ️ Start processing command /merge

2026-07-14 15:12:14 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 2h (p90).

2026-07-14 16:48:08 UTC ℹ️ MergeQueue: This merge request was merged

@gh-worker-dd-mergequeue-cf854d gh-worker-dd-mergequeue-cf854d Bot merged commit 658a3e4 into master Jul 14, 2026
584 checks passed
@gh-worker-dd-mergequeue-cf854d gh-worker-dd-mergequeue-cf854d Bot deleted the daniel.mohedano/karate-v2-instrumentation branch July 14, 2026 16:48
@github-actions github-actions Bot added this to the 1.65.0 milestone Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chatgpt-codex-connector chatgpt-codex-connector[bot] chatgpt-codex-connector[bot] left review comments
@datadog-official datadog-official[bot] datadog-official[bot] left review comments
@dougqh dougqh dougqh approved these changes
@gnufede gnufede gnufede approved these changes
@vandonr vandonr Awaiting requested review from vandonr vandonr is a code owner automatically assigned from DataDog/apm-idm-java

Assignees

No one assigned

Labels

comp: ci visibility Continuous Integration Visibility tag: ai generated Largely based on code generated by an AI or LLM tag: override groovy enforcement Override the "Enforce Groovy Migration" check type: feature Enhancements and improvements

Projects

None yet

Milestone

1.65.0

Development

Successfully merging this pull request may close these issues.

3 participants

@daniel-mohedano @dougqh @gnufede